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ABSTRACT:  The  DoD  is  pursuing  an  end-to-end,  seamless,  network-centric  enterprise  communications 
infrastructure  to  support  a  wide  range  of  operating  conditions  and  network  topologies.  Evaluating  the  achievable 
performance  of  this  communications  infrastructure,  as  it  evolves,  is  essential  to  the  user  community  in  order  to  guide 
their  ongoing  requirements,  design,  and  procurement  activities.  Tactical  edge  applications  present  significant 
challenges  to  network  evaluation  methods  since  they  often  include  mobile  ad-hoc  networks  (MANETs)  that  employ  a 
wide  range  of  platform  types  (ground-based,  air-based,  and  satellite -based),  traffic  types  (data,  voice,  video,  and 
multimedia ),  delivery  methods  (unicast  and  multicast ),  offered  traffic  loads  (kilobits/sec  through  megabits/sec),  and 
numbers  of  nodes  (from  10s  to  1000s).  The  complexity  exhibited  by  tactical  edge  applications  typically  demands  the 
use  of  Modeling  and  Simulation  (M&S)  techniques,  supported  by  high-fidelity  models,  to  adequately  quantify 
achievable  performance  on  an  end-to-end  basis.  However,  these  high-fidelity  models  often  have  very  long  runtimes, 
and  restrictive  limitations  on  scenario  sizing. 

We  investigate  the  application  of  the  DoD  High  Level  Architecture  (HLA)  and  High  Performance  Computing  (HPC) 
platforms  to  address  the  performance  demands  associated  with  analyzing  tactical  edge  applications.  A  federation 
comprised  of  two  Soldier  Radio  Waveform  (SRW)  federates  and  one  Wireless  Network  after  Next  (WNaN)  federate  is 
developed  and  executed  within  an  HPC  environment  at  Aberdeen  Proving  Grounds  (APG).  High-fidelity  OPNET 
models  are  used  to  represent  the  SRW  and  WNaN  waveforms.  Situational  Awareness  (SA)  multicast  traffic  is 
delivered  among  the  nodes  represented  within  each  of  the  three  federates.  Unicast  traffic  is  exchanged  between  the 
SRW  federates,  in  the  presence  of  this  SA  background  traffic,  using  the  WNaN  federate  as  a  transit  network. 
Performance  metrics  include:  run  time,  memory  allocation,  and  achievable  throughput  and  latency  as  a  function  of 
background  SA  traffic  load. 
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1.  Introduction 

The  DoD  has  been  pursuing  a  doctrine  of  network¬ 
centric  warfare  (NCW)  for  the  past  decade  and  a  half 
[1].  And  while  the  appeal  of  using  information 
technology  to  enable  the  robust  networking  of  well- 
informed,  geographically  dispersed  forces  is  clear, 
designing  and  reasoning  about  the  behavior  of 
network-enabled  forces  has  proved  challenging.  A 
particular  challenge  is  reasoning  about  network 
behaviors  in  a  wireless  regime  -  tools  that  support 
the  robust  design  and  evaluation  mobile  ad-hoc 
networks  (MANET)  at  very  high  fidelities  and  very 
large  scales  do  not  yet  exist. 

The  U.S.  Army  Research  Laboratory  (ARL)  Mobile 
Network  Modeling  Institute  (MNMI)  seeks  to 
achieve  a  capability  to  design  and  test  mobile  ad-hoc 
networks  at  the  levels  of  fidelity  and  scale  necessary 
to  understand  the  behaviors  of  NCW  technologies  in 
the  full  range  of  conditions  in  which  they  will  be 
employed.  The  MNMI  is  pursuing  the  use  of  High 
Performance  Computing  (HPC)  as  a  primary  enabler 
for  this  capability. 

In  this  paper  we  describe  the  development  of  an 
initial  modeling  and  simulation  (M&S)  capability  for 
the  MNMI.  The  capability  supports  the  integration 
of  existing  waveform  simulations  running  in  an  HPC 
context.  Section  2  presents  a  brief  overview  of 
MNMI  and  its  near-term  and  long-term  objectives. 
Section  3  describes  the  design  principles  for  a  near- 
term  M&S  framework.  In  Section  4,  we  discuss  the 
development  of  an  instance  of  the  near-term 
framework  using  OPNET  models  of  the  Soldier 
Radio  Waveform  (SRW)  and  Wireless  Network  after 
Next  (WNaN)  waveforms.  Conclusions  and 
directions  for  future  work  are  given  in  Section  5. 


2.  Mobile  Network  Modeling  Institute 
(MNMI) 

The  current  practice  in  reconfigurable  mobile  ad  hoc 
networks  is  to  use  empirical  models,  simulations, 
emulations,  and  experimentations  in  a  stovepipe 
fashion  with  minimal  use  of  high  performance 
computing  (HPC)  resources.  These  evaluations  do 
not  have  the  fidelity  or  the  scalability  to  adequately 
predict  how  large-scale  networks  will  perform  in 
realistic  environments.  Furthermore,  there  is 
currently  no  efficient  way  to  exploit  the  results  of 
each  of  these  approaches  with  a  subsequent 
evaluation. 


To  promote  the  use  of  high  performance  computing 
resources  for  network  modeling,  the  DoD  High 
Performance  Computing  Modernization  Program 
Office  (HPCMOD)  funded  ARL  to  create  the  Mobile 
Network  Modeling  Institute  in  2008.  The  MNMI 
includes  researchers  from  ARL,  Naval  Research 
Laboratory,  the  Communications-Electronics 
Research,  Development,  and  Engineering  Center 
(CERDEC),  MITRE  Corporation,  Program  Executive 
Office  Command  Control  Communications  Tactical 
(PEO  C3T),  Rensselaer  Polytechnic  Institute, 
Kitware,  Stanford  University  and  the  University  of 
Minnesota. 

The  MNMI  vision  is  to  develop  scalable  software 
tools  that  transform  the  ways  in  which  DoD  models, 
simulates,  emulates,  and  experiments  with  dynamic 
reconfigurable  mobile  warfighter  networks. 

The  Institute  seeks  to  exploit  the  power  of  HPC  and 
scalable  software  to  (1)  develop  the  fundamental 
knowledge  required  to  enable  a  priori  prediction  of 
the  behaviors  of  diverse  and  dynamic  networks;  (2) 
understand  the  design  trade-offs  and  impact  of 
various  technologies  under  a  wide  variety  of  dynamic 
adverse  conditions;  and  (3)  quantify  the  impact  of 
network  technologies  both  technically  and 
operationally  to  make  acquisition  decisions. 

To  achieve  this  vision,  the  MNMI  established  several 
objectives: 

•  Develop  and  apply  HPC  software  for  the 
analysis  of  MANETs  in  complex 
environments. 

•  Develop  an  enabling  interdisciplinary 
computing  environment  that  links  models 
throughout  the  Simulation,  Emulation,  and 
Experimentation  (SEE)  cycle. 

•  Leverage  the  powerful  synergistic  relationship 
between  simulation,  emulation,  and 
experimentation. 

•  Expand  DoD  workforce  that  is  cross-trained  in 
computational  software  and  network  science 
skills. 

•  Deliver/support  software  and  train  the  DoD 
HPC  user  community,  and  significantly  extend 
it  to  key  NCW  transformation  programs. 

A  key  component  of  the  MNMI  is  the  linking  of 
simulation,  emulation  &  experimentation.  This  is 
being  done  through  the  development  of  the  Network 
Interdisciplinary  Computing  Environment  (NiCE) 
[2].  NiCE  provides  a  common  data  model  and 
format  that  links  models  throughout  the  simulation 


©201 1 -The  MITRE  Corporation  and  U.S.  Army  Research  Laboratory.  All  rights  reserved.  Approved  for 
Public  Release:  11-0106.  Distribution  Unlimited 


(S),  emulation  (E)  and  experimentation  (E)  cycle  as 
illustrated  in  Figure  1 . 


Figure  1:  Simulation,  Emulation  Experimentation  (SEE)  Cycle. 


At  the  core  of  NiCE  is  the  cross -platform  extensible 
discrete-event  mobile  Networking  Data  Model  and 
Format  (NetDMF)  [3].  NetDMF  is  a  superset  of  the 
extensible  Data  Model  and  Format  (Xdmf).  Xdmf  is 
used  worldwide  in  physics  based  simulations  such  as 
computational  chemistry,  structural  mechanics  and 
fluid  mechanics.  NetDMF  extends  Xdmf  to  include 
network  and  mobility  features  needed  to  support  the 
modeling  of  mobile  networks.  Using  NiCE, 
researchers  can  leverage  visualization  and  analysis 
tools  across  all  elements  of  the  cycle. 

The  DoD  has  made  a  large  investment  in  developing 
radio  waveforms  using  existing  discrete  event 
simulators  (DES)  such  as  OPNET  and  Qualnet. 
Given  the  amount  of  effort  invested  in  developing 
and  validating  these  models,  the  Institute  decided  it 
would  be  infeasible  and  unrealistic  to  start  from  a 
clean  slate.  Instead,  the  Institute  decided  to  break  the 
modeling  and  simulation  effort  into  near-term  and 
long-term  thrusts. 

The  near-term  thrust  seeks  to  demonstrate  the  value 
of  the  SEE  cycle  approach  to  Institute  stakeholders 
by  using  existing  legacy  waveform  models  and  DES 
tools  with  minimal  changes  and  installing  them  in  the 
HPC  environment.  While  these  tools  and  models 
cannot  fully  exploit  the  HPC  environment,  they  can 
demonstrate  that  the  vision  is  achievable  and  provide 
the  stakeholders  with  a  significantly  enhanced 
analysis  capability.  A  detailed  discussion  of  the 
near-term  thrust  is  found  in  Section  3. 

The  longer  term  thrust  is  to  develop  a  massively 
parallelizable  discrete  event  simulator  that  can  utilize 
the  HPC  resources  more  effectively. 


Figure  2  shows  how  the  Institute’s  M&S  capabilities 
are  evolving  over  time.  The  Institute  initially  started 
out  with  individual  radio  waveform  models  operating 
in  a  single  DES  running  on  a  single  processor.  The 
Institute  is  currently  running  multiple  radio 
waveform  models  on  multiple  processors  still  within 
a  single  DES.  The  next  planned  capability 
improvement  is  running  multiple  radio  waveform 
models  developed  for  use  on  different  DES  engines. 


Figure  2:  MNMI  M&S  migration  path. 


3.  Near-Term  M&S  Framework 

The  tyranny  of  scales  is  well-known  in  distributed 
systems.  Solutions  that  scale  effectively  to  tens  of 
processing  elements  (in  today’s  parlance:  cores), 
often  do  not  scale  to  hundreds.  Similarly,  solutions 
that  scale  effectively  to  hundreds  of  cores  may  not 
scale  to  thousands,  nor  thousands  to  tens  of 
thousands,  and  so  on.  These  limitations  are  a 
function  of  well-known  issues  in  synchronization, 
communication  and  fault  tolerance,  and  the 
availability  of  exploitable  parallelism  ala  Amdahl’s 
Law  [4]  or  of  a  sufficiently-sized  workload  ala 
Gufstafson’s  Law  [5]. 

While  petascale  and  exascale  computing  on  systems 
consisting  of  hundreds  of  thousands  of  cores  is  the 
scientific  frontier  [6],  the  figure  of  merit  for  the 
MNMI  is  the  provision  of  tools  that  can  effectively 
scale  to  use  the  resources  available  in  today’s 
“commodity”  HPC  systems,  consisting  of  tens  of 
thousands  of  cores. 

To  date,  only  a  few  MANET  modeling  packages 
have  demonstrated  the  ability  to  approach  today’s 
HPC  scales  (see  e.g.,  [7,  8]).  These  systems  are 
based  on  well- studied  algorithms  in  parallel  discrete 
event  simulation  (PDES)  [9]  and  often  have  been 
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finely  tuned  to  match  the  architecture  of  the 
underlying  HPC  hardware  [10,  11].  MNMI  is 
evaluating  extant  PDES -based  MANET  modeling 
packages  for  their  suitability  to  serve  as  its  long-term 
M&S  framework,  with  specific  attention  to  two 
open-source  frameworks:  the  Rensselaer  Optimistic 
Simulation  System  (ROSS)  [7],  and  ns-3  [12].  Once 
the  long-term  framework  is  selected,  significant 
effort  is  expected  to  be  required  to  populate  the 
framework  with  the  waveform  models  necessary  to 
make  it  a  useful  analytical  capability.  Part  of  this 
effort  will  be  undertaken  within  the  auspices  of  the 
MNMI,  but  much  of  the  effort  will  need  to  be 
accomplished  through  other  programs. 

In  the  interim,  the  objectives  of  the  MNMI  include 
the  development  of  a  near-term,  HPC-based 
analytical  capability  based  on  waveform  models  that 
exist  today. 

Since  a  great  many  models  of  future  waveforms  have 
been  developed  in  the  OPNET  simulation 
environment,  and  since  OPNET  provides  -  and 
actively  supports  -  a  High  Level  Architecture  (HLA) 
interface  for  its  product  line,  an  obvious  choice  for 
the  interim  solution  is  “OPNET-over-HLA”.  Such  a 
solution  potentially  enables:  (1)  the  analysis  of 
heterogeneous  (multi-waveform)  networks  by 
federating  models  of  different  waveforms,  and  (2)  the 
scaling  of  single  waveform  models  by  splitting  the 
model  into  multiple  federates. 

3.1  Near-term  Federation  Object  Model  (FOM) 

Given  the  general  approach  of  federating  OPNET 
simulations,  we  need  to  answer  the  question,  at  what 
level(s)  in  the  network  protocol  stack  can  runtime 
interoperation  be  supported  and  how  does  this  choice 
constrain  the  kinds  of  scenarios  and  analyses  that  can 
be  run? 

We  observe  that  for  objective  (1)  above  -  the 
analysis  of  multi-waveform  networks  where  each 
waveform  is  encapsulated  within  a  single  federate  - 
interoperation  at  the  network  layer,  specifically  via 
the  exchange  of  Internet  Protocol  (IP)  packets 
between  federates,  is  sufficient  for  many  network 
design  and  analysis  problems.  In  many  operational 
contexts,  and  for  most  future  waveforms,  geographic 
and  frequency  separation  limits  the  need  to  represent 
interactions  at  the  physical  and  data  link  layers.  On 
the  other  hand,  this  assumption  does  not  hold  for 
objective  (2)  above  -  scaling  single  waveform 
models  by  splitting  them  into  multiple  federates. 
Here,  interactions  at  the  physical  and  Media  Access 
Control  (MAC)  layers,  which  are  represented  in  the 
standalone  OPNET  model,  must  generally  be 
preserved  as  the  model  is  broken  into  federates. 


In  the  spirit  of  “you  have  to  start  somewhere”  and 
“crawl  before  you  walk”,  we  adopt  an  initial 
approach  based  on  IP  packet  interoperation  (similar 
to  the  approaches  presented  in  [13,  14])  and  leave 
MAC  and  physical  layer  interoperation  for  future 
work. 

3.2  HLA  scaling 

As  we  note  previously,  the  long-term  M&S 
Framework  for  the  MNMI  is  anticipated  to  be  in  the 
form  of  a  parallel  discrete  event  simulation  (PDES) 
engine.  And  it  is  the  long-term  M&S  framework  that 
must  realize  HPC  scales.  The  scalability  of  the  near- 
term  HLA-based  framework  is  of  lesser  importance 
than  its  analytical  value.  However,  we  are  still 
interested  in  understanding,  and  maximizing,  the 
general  scalability  of  the  near-term  approach. 

The  performance  of  the  commercially  available  HLA 
Runtime  Infrastructures  (RTIs)  is  generally  well- 
studied  (see,  e.g.,  [15,  16]).  These  RTIs  provide  the 
full-range  of  HLA  services  and  are  generalized  to 
operate  in  any  Ethernet-based  network.  While  the 
performance  advantages  of  supporting  special- 
purpose  computing  architectures  and  very  high  speed 
networking  fabric  have  been  demonstrated  [17,  18, 
19],  the  business  case  for  their  support  in  commercial 
contexts  remains  an  open  question  [20,  21]. 

The  target  computing  platform  for  the  MNMI  near- 
term  M&S  framework  is  Harold,  an  SGI  Altix  ICE 
8200  containing  10,752  compute  cores,  32  TB 
memory  and  a  peak  system  performance  of  120 
Teraflops.  The  compute  cores  are  arranged  into 
1,344  compute  nodes  each  with  dual  quad-core  Intel 
Xeon  Nehalem-EP  2.8  GHz  processors.  The 
compute  nodes  are  connected  via  4  Gbps  infiniband 
[22]. 

Based  on  the  benchmarking  approach  suggested  by 
Knight  et  al. ,  we  conducted  a  series  of  latency  and 
throughput  evaluations  of  three  commercially 
available  RTIs  running  on  Harold.  We  attempted  to 
configure  each  RTI  to  achieve  maximum 
performance  for  each  benchmark.  However,  since 
our  analysis  was  conducted  without  active  detailed 
engagement  from  the  RTI  vendors,  we  avoid  making 
any  claims  regarding  the  definitive  performance  of 
the  RTIs  here. 

The  results  of  a  simple  time  advance  benchmark  are 
given  in  Figure  3  below. 
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Figure  2:  An  RTI  performance  benchmark.  Time  Advance 
Grants  (TAGs)  per  second  as  the  number  of  federates  in  the 
federation  increases. 


In  this  test,  we  examine  the  number  of  time  advance 
grants  (TAGs)  per  second  that  an  RTI  can  generate  in 
the  presence  of  some  number  of  federates,  each  of 
which  is  attempting  to  advance  time  as  rapidly  as 
possible.  Essentially,  this  benchmark  evaluates  the 
scalability  of  the  underlying  Lower  Bound  Time 
Stamp  (LBTS)  calculation.  We  scale  from  5  to  100 
federates  and  observe  that  two  of  the  commercial 
RTIs  demonstrate  similar  results,  ranging  from 
thousands  of  T AGs/sec  for  5  federates  to  hundreds  of 
T AGs/sec  for  100  federates.  The  third  exhibits  a  flat 
behavior,  providing  around  100  T AGs/sec  for  any 
sized  federation. 

For  sake  of  comparison  with  the  results  reported  for 
“high  performance”  RTIs  (e.g.,  [17,  18,  19])  we 
adapted  ROSS  to  provide  minimal  RTI  capabilities, 
namely:  (1)  support  for  conservative  time 

synchronization  (see  [9]  for  a  discussion  of  adding 
conservative  synchronization  to  an  optimistic 
simulation  engine),  and  (2)  support  for  HLA  services 
to  create/destroy,  publish/subscribe,  send/receive, 
and  advance  time.  Running  the  same  benchmark,  we 
observe  that  this  PDES -based  minimal  RTI  is  able  to 
generate  500,000  TAGs/sec  for  5  federates,  scaling 
to  52,000  TAGs/sec  for  100  federates.  We 
conjecture  that  primary  reasons  for  the  superior 
performance  of  the  ROSS-based  RTI  are:  (1) 
omission  of  the  full  range  of  HLA  services;  (2) 
pointer-based  implementation  using  the  high- 
performance  Message  Passing  Interface  (MPI)  [23]; 
(3)  tailorability  to  Harold  architecture;  and  (4) 
highly-optimized  LBTS  algorithm  (based  on  ROSS 
Global  Virtual  Time  (GVT)  algorithm). 

Since  a  desire  for  the  near-term  M&S  framework  is 
to  provide  a  usable,  supportable,  analytical 
capability,  and  since  the  performance  demands 
required  to  support  an  IP -packet-based  FOM  are 
expected  to  be  reasonably  low,  the  MNMI  is 
currently  pursuing  a  commercial  solution.  Our 
efforts  to  date  have  been  conducted  using  the  RTI 


NG  Pro  software  from  Raytheon VTC.  As  we  begin 
to  address  MAC  and  physical  layer  interoperability — 
potentially  resulting  in  vastly  greater  numbers  of 
interactions  among  federates — and  if,  and  as,  the 
number  of  federates  in  our  federations  scales  beyond 
the  hundreds,  the  need  to  revisit  PDES -based  RTI 
support  for  the  near-term  framework  may  arise. 

3.2  Adapting  OPNET  models  for  HLA 

As  noted  above,  we  adopt  a  basic  approach  to 
federating  wireless  network  models  at  the  IP  layer 
similar  to  those  described  in  [13,  14].  Within  each 
OPNET  simulation,  we  simply  add  an  “HLA  node” 
which  serves  as  a  synchronizing  agent  and  a  gateway 
for  sending  and  receiving  IP  packets  to  nodes  in 
other  federates.  While  adding  this  HLA  node 
supports  conservative  time  synchronization  across 
the  OPNET  federates,  with  very  little  perturbation  of 
the  original  model  code,  this  basic  modification  does 
not  explicitly  transfer  any  data  between  models. 
Additional  work  is  required  in  order  to  correctly 
route  desired  traffic  from  one  network  model  to 
another.  Unfortunately,  there  is  no  general  solution 
for  this;  rather  it  requires  understanding  the  existing 
model  designs,  and  adding  customized  interfacing 
nodes  in  each  model  to  intercede,  encode,  send, 
receive,  decode,  and  utilize  traffic  appropriately.  A 
case  study  is  described  in  Section  4  below. 

Once  data  arrives  at  an  HLA  routing  node,  the  actual 
transmitting  of  data  bytes  (in  this  case  a 
representation  of  an  OPNET  IP  packet)  from  the 
OPNET  HLA  node  to  the  HLA  network  is  trivial 
compared  to  the  other  design  steps.  This  data 
transmission  requires  the  definition  of  a  small  HLA 
Federation  Object  Model  (FOM)  in  order  to  send 
data  across  HLA.  The  OPNET  mechanism  maps 
HLA  FOM  elements  to  a  customized  OPNET  packet 
definition.  Neither  OPNET  nor  HLA  attempt  to 
translate  this  data;  part  of  the  custom  HLA  routing 
code  must  interpret  the  data  and  utilize  appropriately. 

3.3  Executing  OPNET/HLA  in  a  batch 
environment 

Many  (most)  HLA  federations  operate  in  an 
interactive  mode,  where  an  operator  interacts  with  a 
federate  or  management  application  to  run  and  pause 
the  federate/federation.  Our  system  is  intended  for 
batch  submission  on  remote  HPC  assets,  so  relying 
on  such  interactive  means  for  control  is  infeasible. 

When  operating  single  OPNET  simulations  with 
HLA  enabled,  the  first  OPNET  federate  to 
successfully  join  (unless  designed  otherwise)  will 
immediately  begin  advancing  time  in  accordance 
with  OPNET’ s  internal  event  list. 
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In  order  to  orchestrate  our  group  of  federates  and  to 
prevent  the  above  condition,  we  developed  and 
introduced  a  controlling  “pacer”  federate.  This 
federate  joins  first  and  forms  the  federation;  sets  time 
management  features  such  that  it  participates  in  the 
time  advancement  decisions  (time -regulating  in  HLA 
parlance),  then  remains  at  time  =  t0  until  the  expected 
number  of  OPNET  federates  have  joined.  Once  this 
condition  is  achieved,  the  pacer  federate  removes  the 
time-regulating  setting,  triggering  the  federation  to 
proceed  forward  with  all  OPNET  federates 
advancing  in  synchronized  fashion  based  on  the 
modeling  activities. 

4.  Case  Study  -  A  Multi- Waveform 
Prototype 

Two  key  waveforms  under  development  by  the  DoD 
to  significantly  enhance  warfighter  communications 
at  the  tactical  edge  include  the  Soldier  Radio 
Waveform  (SRW)  [24]  and  the  Wireless  Network 
after  Next  (WNaN)  waveform  [25]. 

SRW  is  being  developed  through  the  Joint  Tactical 
radio  System  (JTRS)  program  to  provide  mobile  ad- 
hoc  network  (MANET)  voice  and  data  for 
dismounted  soldiers  and  small  form  factor 
applications.  SRW  is  intended  to  form  stub/leaf 
networks  with  limited  transit  capability  to  other  SRW 
networks  and  will  rely  upon  other  waveforms  for 
backbone  transport  service. 

The  Defense  Advanced  Research  Projects  Agency 
(DARPA)  developed  the  Wireless  Network  after 
Next  (WNaN)  waveform  to  support  inexpensive 
warfighter-portable  multi-hop  radios  that  support 
spectrum  agility,  dynamic  spectrum  access  (DSA), 
and  disruption  tolerant  networking  (DTN). 

For  the  purposes  of  this  effort,  the  WNaN  waveform 
is  used  to  provide  the  backbone  transport  service  for 
two  separate  SRW  leaf  networks.  The  SRW  OPNET 
model  used  in  the  multi-waveform  simulation  was 
developed  by  the  U.S.  Army  Communications 
Electronic  Research  &  Development  Center 
(CERDEC)  and  Materiel  Systems  Analysis  Activity 
(AMSAA)  while  the  WNaN  OPNET  model  was 
developed  by  BBN  Technologies. 

4.1  A  simple  multi- waveform  architecture 

The  top  level  architecture  of  the  Multi -Waveform 
Tri-Federate  Capability  is  depicted  in  Figure  4. 
Although  unicast  traffic  may  be  sent  and  received 
within  any  federate,  only  the  SRW  federates  are  the 
source  and  destination  of  unicast  traffic.  Within  each 
SRW  federate,  routing  is  handled  by  the  DS  Routing 


module  which  is  a  component  of  SRW  waveform  and 
resides  below  the  IP  layer.  If  the  destination  of  a 
traffic  packet  is  in  the  same  federate  as  the  source  of 
the  packet,  DS  Routing  module  routes  this  packet  to 
its  destination.  However,  if  the  destination  belongs 
to  another  SRW  federate,  the  DS  Routing  module  in 
each  SRW  node  considers  the  destination  “unknown” 
and  routes  the  packet  to  the  gateway  as  a  default 
route. 


Federate  1  Federate  2 


Figure  3:  Top  Level  Architecture  of  the  Multi -Waveform  Tri- 
Federate  Capability. 


When  the  packet  is  received  by  the  SRW  radio 
located  at  the  gateway  node,  it  recovers  the  original 
IP  packet  and  forwards  it  on  its  host  interface.  The 
subnet  model  representing  the  gateway  has  been 
modified  to  include  an  HLA/RTI  Handler.  A  packet 
with  a  destination  outside  the  federate  within  which  it 
is  generated  is  received  by  the  HLA/RTI  Handler  for 
transmission  to  its  intended  federate.  The  HLA/RTI 
Handler  receives  each  of  these  IP  packets,  serializes 
them  for  transmission  through  the  HLA/RTI 
environment,  and  delivers  the  packet  to  the  HLA/RTI 
environment.  This  causes  all  the  HLA  federates  to 
receive  a  copy  of  the  HLA  packet.  The  serialized 
packet  includes  the  source  and  destination  federate 
identifiers  (IDs)  which  are  used  to  determine  the 
intended  receiver  federate  of  the  packet.  If  the 
source  federate  ID  is  either  1  or  2,  the  packet  is 
received  by  federate  3  (the  WNaN  federate)  which 
acts  as  a  transit  network  between  the  two  SRW 
federates. 

In  the  WNaN  federate,  an  HLA/RTI  Handler  has 
been  added  to  the  WNaN  network  model,  to  process 
packets  from/to  the  HLA/RTI  environment.  For  each 
external  federate,  an  internal  gateway  node  within  the 
WNaN  network  is  identified  as  the  entry  point  for 
packets  to/from  that  federate.  The  gateway  node  is 
an  “enhanced”  version  of  the  WNaN  node/radio 
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model,  with  an  additional  “externaljxaffic”  module 
above  IP  to  process  the  external  IP  traffic  received 
from  other  federates.  This  “extemal_traffic”  module 
takes  an  IP  packet  sent  from  another  federate, 
determines  the  destination  gateway  within  the  WNaN 
network/subnet,  based  on  a  set  of  destination  federate 
information,  and  sends  the  packet  as  the  payload  of  a 
WNaN  IP  packet  to  the  that  gateway’s  IP  address.  At 
the  destination  gateway  node,  the  “external_traffic” 
module  takes  the  original  IP  packet  and  sends  it  to 
the  HLA/RTI  Handler  for  processing  and 
subsequently  to  the  HLA/RTI  environment. 

When  the  WNaN  federate  acts  as  the  transit  network, 
the  HLA/RTI  Handler  in  the  WNaN  federate  receives 
the  serialized  IP  packet  and  reconstructs  the  original 
IP  packet.  Since  each  federate  has  a  gateway 
associated  with  it,  the  packet  is  delivered  to  the 
appropriate  gateway  for  transmission  through  the 
WNaN  network  to  another  gateway  that  is  collocated 
with  the  destination  SRW  federate. 

When  a  packet  is  received  by  the  WNaN  gateway 
that  is  collocated  with  destination  SRW  federate,  it 
forwards  the  payload  IP  packet  to  the  WNaN  HLA 
Handler,  which  serializes  the  packet  and  delivers  it  to 
the  HLA/RTI  environment.  For  this  transmission, 
the  source  federate  ID  is  set  to  3  (ID  of  WNaN 
federate),  but  the  original  federate  ID  is  maintained. 
The  source  and  destination  node  ID  and  federate  ID 
of  a  node  are  derived  from  the  IP  address  assigned  to 
a  node. 

When  a  serialized  IP  packet  is  received  by  an  SRW 
federate,  it  determines  if  it  is  the  destination  federate 
and  if  the  source  of  the  packet  is  the  WNaN  federate. 
If  so,  it  routes  the  packet  via  its  internal  network  DS 
routing  module.  Otherwise,  it  ignores  the  packet. 

Situational  Awareness  (SA)  traffic  can  be  generated 
within  any  federate  as  multicast  traffic.  All  multicast 
packets  are  routed  to  destinations  within  the  federate. 
Multicast  packets  are  not  forwarded  to  the  HLA/RTI 
Handlers  for  delivery  to  other  federates. 

4.2  Multi-waveform  scenario 

Two  multi- waveform  scenarios  were  developed  as 
part  of  this  effort  and  are  referred  to  as  the  9-9-9  and 
42-75-42  configurations,  respectively.  Figure  5 
provides  a  high-level  portrayal  of  both 
configurations.  The  9-9-9  configuration  includes  9 
nodes  in  each  of  the  three  federates  whereas  the  42- 
75-42  configuration  includes  42  nodes  in  each  of  the 
SRW  federates  and  75  nodes  in  the  WNAN  federate. 


SRW  Fed  1  SRW  Fed  2 


Figure  4:  High-Level  Mult-Federate  Configuration. 

Varying  levels  of  multicast  situational  awareness 
(SA)  traffic  is  generated  within  each  federate  for  both 
configurations.  In  addition  to  this  SA  traffic,  a 
unicast  traffic  flow  is  sent  from  SRW  federate  1  to 
SRW  federate  2,  using  the  WNAN  federate  as  a 
transit  network.  The  complete  set  of  offered  traffic 
loads  for  both  configurations  is  shown  in 
Table  1. 

Both  scenario  configurations  were  positioned  in  the 
Fort  Dix  area.  The  Terrain  Integrated  Rough  Earth 
Model  (TIREM)  [26]  was  used  to  generate  the  path 
attenuation  data  between  each  pair  of  nodes  within  a 
federate  -  there  was  no  connectivity  between 
federates  other  than  through  the  gateway  nodes. 

4.3  Execution  in  HPC  environment 

As  noted  Section  3.3,  executing  an  HLA  federation 
in  an  HPC  environment  offers  a  few  challenges  not 
typically  found  in  “traditional”  HLA  applications. 
HPC  execution  is  predominantly  a  non-interactive 
activity,  typically  initiated  by  a  user  submitting  a 
scripted  job  for  execution,  and  allowing  the  scripting 
mechanism  to  determine  where  and  when  to 
eventually  execute  the  job.  For  our  ARL  host  HPC 
system,  Harold  [22],  this  scripting  feature  is  provided 
by  the  “Portable  Batch  System”  PBS  [27]. 

For  this  federation,  we  developed  a  submission  script 
containing  7  jobs: 

1  -  main  launching  job,  utilizing  the  PBS 
“job_array”  feature  to  cause  N  iterations  of  the 
script  to  run,  with  increasing  index  numbers; 

2  -  RTI  executive  process;  job  start  after  job  1 
starts 

3  -  Pacing  federate;  job  start  after  job  2  starts 

4  -  SRW#1  model; 

5  -  SRW#2  model; 

6  -  WNaN  model; 
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5  thru  7  depend  on  job  3  starting  successfully; 
also  made  “co-dependent”  so  they  only  start 
when  they  all  can  start. 

7  -  shutdown  process;  only  starts  after 

completion  of  3,4,5,  and  6.  This  job  includes 
post-cleanup  and  collection  of  output  files. 

We  observe  a  few  challenges  to  operating  in  this 
environment: 

•  Little  support  for  RTIexec  in  non-inter  active 
mode.  There  is  no  clean  remote  “shutdown” 
mechanism  for  the  executive  once  it  is 
started.  We  rely  on  PBS  job  queue  delete  to 
remove  a  running  RTI  executive  process. 

•  Wide  variance  in  job  execution  time.  Often, 
one  of  seven  jobs  in  a  set  started,  but  others 
remained  queued  for  many  hours.  The 
alloted  time  for  the  entire  job  set  expired 
before  full  set  could  complete.  This 
required  re-evaluation  and  adjustment  of  job 
submission  dependencices. 

•  Job  node  selection  and  control  use  MPI 
feature.  PBS  features  to  specifying  node 
resources  are  primarily  oriented  towards 
MPI  jobs,  and  are  not  easily  applied  in  our 
context. 


4.4  Results 

We  begin  this  section  with  an  important  disclaimer: 
the  purpose  of  generating  performance  results  for  the 
two  configurations  under  investigation  is  to 
demonstrate  the  utility  of  the  tri-federate  capability 
developed  in  this  effort.  The  results  presented  should 
not  be  used  to  evaluate  the  capabilities  of  the  SRW 
and/or  WNAN  waveforms. 

We  define  24  configurations  for  runs  made  using  the 
Harold  high  performance  computing  (HPC)  platform, 
as  listed  in  Table  1.  The  unicast  traffic  flow  period 
varies  from  1  second  to  .01  seconds  (i.e.,  packet  rate 
of  1  packet/sec  to  100  packets/sec),  with  a  packet 
size  of  1024  bytes.  The  SA  traffic  periods  are  chosen 
to  stress  the  network  performance  across  the  range  of 
unicast  traffic  levels  investigated.  Three  levels  of  SA 
traffic  are  investigated  for  each  configuration  and  are 
referred  to  as  “low”,  “medium”  and  “high”.  The 
SRW  SA  packet  size  used  is  30  bytes  while  the 
WNAN  SA  packet  size  is  56  bytes.  The  resulting  SA 
traffic  loads  for  the  9-9-9  and  42-75-42 
configurations  are  shown  in  6  and  7,  respectively. 


Low  Medium  High 

SA  Offered  Traffic  Load  Configuration 


Figure  6:  SA  Offered  Traffic  Load  for  the  9-9-9  Configuration. 


Figure  7 :  SA  Offered  Traffic  Load  for  the  42-75-42 
Configuration. 


The  primary  figure  of  merit  measured  with  the  tri- 
federate  capability  developed  in  this  effort  is  the 
achievable  message  completion  rate  (MCR).  The 
MCR  is  measured  as  a  function  of  the  offered  unicast 
SRW  federate  1— >  SRW  federate  2  traffic  load  for 
the  “low”,  “medium”,  and  “high”  SA  traffic  levels. 
The  MCR  performance  results  generated  for  the  9-9- 
9  and  42-75-42  configurations  are  shown  in  Figure  8 
and  Figure  9,  respectively.  As  shown,  the  achievable 
MCR  decreases  for  both  configurations  as  the  SA 
background  traffic  load  increases  and  as  the  unicast 
traffic  load  increases. 

There  are  a  number  of  useful  applications  for  the 
multi-federate  capability  developed  in  this  effort. 
For  example,  this  capability  could  be  used  to  help 
define  the  minimum  SA  traffic  period  possible  in  a 
muilti-waveform  network  within  the  context  of 
achieving  a  given  MCR  for  alternative  user 
application  traffic  profiles.  Another  application 
could  be  to  compare  the  capabilities  of  alternative 
transit  network  waveforms  (e.g.,  WNAN,  WNW, 
etc.)  in  the  presence  of  varying  traffic  profiles  and 
user  configurations.  The  capability  can  be  readily 


©201 1 -The  MITRE  Corporation  and  U.S.  Army  Research  Laboratory.  All  rights  reserved.  Approved  for 
Public  Release:  11-0106.  Distribution  Unlimited 


extended  to  support  applications  requiring  an 
increased  number  of  federates  as  well  as  waveforms 
where  existing  models  exist  to  represent  their 
behavior. 


SRW  Fedl->Fed2  Unicast  Offered  Load  (pps) 
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Figure  8:  MCR  as  Function  of  Unicast  Offered  Traffic  Load  for 
9-9-9  Configuration. 
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Figure  9:  MCR  as  Function  of  Unicast  Offered  Traffic  Load  for 
42-75-42  Configuration. 

5.  Conclusions 

The  MNMI  seeks  to  exploit  the  power  of  HPC  and 
scalable  software  to:  (1)  develop  the  fundamental 
knowledge  required  to  enable  a  priori  prediction  of 
the  behaviors  of  diverse  and  dynamic  networks;  (2) 
understand  the  design  trade-offs  and  impact  of 
various  technologies  under  a  wide  variety  of  dynamic 
adverse  conditions;  and  (3)  quantify  the  impact  of 
network  technologies  both  technically  and 
operationally  to  make  acquisition  decisions. 

While  the  long-term  software  solutions  to  achieve 
these  goals  will  likely  require  a  reformulation  of  the 
waveform  models  that  exist  today,  we  have 
developed  a  near-term  capability  that  allows  users  to 


exploit  HPC  capabilities  using  existing  waveform 
models  today.  The  MNMI  will  continue  to  evolve 
this  near-term  capability  as  it  pursues  its  longer-term 
objectives. 
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Table  1:  Offered  Traffic  Loads  for  the  9-9-9  and  42-75-42  Configurations. 


Run# 

Model 

Configuration 

# nodes 

SA  packet 
size  (bytes) 

SA  period 
(sec) 

Unicast 
packet  size 
(bytes) 

Unicast  period 
(sec) 

WNAN 

Transit  Federation 

75 

56 

3.1 

1 

SRW 

Source& 

Destination 

Federations 

42 

30 

5 

1024 

1 

2 

0.1 

3 

0.05 

4 

0.01 

WNAN 

Transit  Federation 

75 

56 

3.1 

5 

SRW 

Sou  rce  & 
Destination 
Federations 

42 

30 

0.5 

1024 

1 

6 

0.1 

7 

0.05 

8 

0.01 

WNAN 

Transit  Federation 

75 

56 

3.1 

9 

SRW 

Sou  rce  & 
Destination 
Federations 

42 

30 

0.3 

1024 

1 

10 

0.1 

11 

0.05 

12 

0.01 

WNAN 

Transit  Federation 

9 

56 

1 

13 

SRW 

Sou  rce  & 
Destination 
Federations 

9 

30 

1 

1024 

1 

14 

0.1 

15 

0.05 

16 

0.01 

WNAN 

Transit  Federation 

9 

56 

0.06 

17 

SRW 

Sou  rce  & 
Destination 
Federations 

9 

30 

0.4 

1024 

1 

18 

0.1 

19 

0.05 

20 

0.01 

WNAN 

Transit  Federation 

9 

56 

0.03 

21 

SRW 

Sou  rce  & 
Destination 
Federations 

9 

30 

0.07 

1024 

1 

22 

0.1 

23 

0.05 

24 

0.01 

©201 1 -The  MITRE  Corporation  and  U.S.  Army  Research  Laboratory.  All  rights  reserved.  Approved  for 
Public  Release:  11-0106.  Distribution  Unlimited 


